Skip to content

[codex] #88 支持 dogfood answer facets - #95

Merged
catoncat merged 1 commit into
mainfrom
codex/issue-88-answer-facet-dogfood
Jul 3, 2026
Merged

[codex] #88 支持 dogfood answer facets#95
catoncat merged 1 commit into
mainfrom
codex/issue-88-answer-facet-dogfood

Conversation

@catoncat

@catoncat catoncat commented Jul 3, 2026

Copy link
Copy Markdown
Owner

Mainline Intent

Intent: int_3266cd23
Status: proposed
Title: 支持 dogfood answer facets

What changed

为私有 dogfood acceptance flow 增加 expected.answerFacets、facet/assertion 分离评分、失败分类、runner/README/scorecard 输出,以及 passing/failing facet 测试覆盖。

Why

#88 要求 dogfood case 不只验证 session/cwd/snippet,还要能验证检索到的 evidence 是否支撑真实回答 facets,并把 failure 归类成可行动的问题层级。

Decisions

  • answer facet schema 位置: 在 expected.answerFacets 下加入 label、mustContain、可选 failureClass (answer facet 是对 retrieved evidence 的预期,不是 find option;放在 expected 可与现有 session/cwd/context assertions 共存。)
  • facet 和既有断言如何评分: DogfoodEvaluation 增加 assertionMark、facetMark、failureClasses,同时保留总 mark/blocking (runner 可分别报告 facet pass/fail 和 session/cwd/match/context assertions,同时 hard/candidate gate 仍使用总体 mark。)
  • failure classification 的表达: 新增枚举 coverage_index、skill_guidance、cli_recall_ranking_context、stale_golden、unclear_case,并附在失败 predicate/facet 上 (Answer-facet dogfood gate v1 #88 需要失败能落到可行动层级;默认 recall/context 问题归 cli_recall_ranking_context,无命中归 coverage_index,facet 可显式覆盖。)
  • session-level evidence read: runner read-range 对 matchSeq 缺失的 hit 使用 --query 回退,并使用 sessionRef 读跨 source session (answer facets 必须绑定真实 evidence;session-level hits 不能因为缺 seq 而跳过上下文读取。)
  • review 后修复: 抽出 buildDogfoodScoreboard,并让 README predicate 输出显示 answer_facet: (独立 Standards review 指出 scoreboard 重复和 answer_facet.answer_facet 可读性差;已修。)

Subsystems: eval, dogfood, acceptance-gate, agent-recall, perf


Summary by cubic

Adds answer facet verification to dogfood evals and acceptance, with separate scoring for assertions vs evidence-backed facets, plus failure classification and updated scoreboards. Satisfies #88 by checking that retrieved evidence supports real answer facets and by grouping failures into actionable classes.

  • New Features
    • Schema: add expected.answerFacets[] with label, mustContain[], optional failureClass.
    • Scoring: compute assertionMark and facetMark in addition to overall mark; keep hard/candidate blocking on overall mark.
    • Failure classes: support coverage_index, skill_guidance, cli_recall_ranking_context, stale_golden, unclear_case; aggregate to failureClasses.
    • Scoreboard: new assertionPass/Fail and facetPass/Fail; buildDogfoodScoreboard used in acceptance gate and runner; perf bench parser updated.
    • Output: README/scorecard show assertion vs facet marks, failure classes, and predicate labels like answer_facet:<label>.
    • Context read: desiredContextMode triggers on facets; for read-range fallback to --query when matchSeq is missing; use sessionRef for read-range and read-page.
    • Tests: added passing/failing facet cases and acceptance-gate coverage.

Written for commit a3c6517. Summary will update on new commits.

Review in cubic

@catoncat
catoncat merged commit c94edbf into main Jul 3, 2026
2 checks passed
@catoncat
catoncat deleted the codex/issue-88-answer-facet-dogfood branch July 3, 2026 06:02
@catoncat catoncat mentioned this pull request Jul 3, 2026
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant